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SUBSTITUTE SPECIFICATION 

TITLE OF THE INVENTION 

METHOD AND DEVICE FOR INPUTTING CONTROL INFORMATION IN COMPUTER 

SYSTEMS 

[0001] This application is t>ased on and hereby claims priority to PCT Application No. 
PCT/EPOO/05917 filed on June 26, 2000, and European Application No. 991 12287.0 filed June 
25. 1999 in Europe, the contents of which are hereby incorporated by reference. 
BACKGROUND OF THE INVENTION 

[0002] -me invention relates to a method and a device for inputting control Infomiaton in 
computer systems, and in particular to an input device and a pertaining method for operating a 
mobile multimedia communication terminal (mobile videophone). 

[00031 Mobile communication terminals such as mobile phones for example increasingly have 
access to "online services". In particular the introduction of the future broadband UMTS mobile 
communication network wll enable high-quality mobile Internet access. 

[0004] Ukewise, ultra-small computer systems (such as palmtop computers for example) on 
which a wide variety of organizer functions (such as address management, diary, task list, eta 
for example) are Implemented are becoming increasingly important 

[0005] The operation of these display-driven computer systems requires an input device with 
which the infomiation displayed on a display unit can be controlled or selected. 

[00061 Mobile ultra-small computer systems of this type usually have so-called "touch 
screens- as an Input device, wherein the display unit is equipped with a touch-sensitive layer 
and it is accordingly possible either to input control infbmfiation or select the displayed 
information by an additional input stylus. The disadvantage of such Input devices however is 
that the additional input stylus has to be accommodated in the divice and consequently may get 
lost Moreover, user Interfaces for these "Touch-screen" display units ai^ mechanically not very 
robust. 

[0007] So-called "mouse" Input devices are also known, wherein a rolling ball is converted Into 
a corresponding electrical signal which is displayed as a pointer symbol or selection information 
on a display unit In addition, control keys can be used as an Input device for computer 
systems. 
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[0008] Particularly in the case of ultra-sman computer systems however, input devices of this 
type are not possible because of the greater amount of space required. 

[00091 As an alternative to the touch-controlled input devices described above, so-called non- 
contact input devices are known in which, for example, a video camera reconJs a position of an 
input stylus, evaluates It. and on the basis of the evaluation a pointer symbol or selection 
infonnation displayed on the display unit is subsequently displaced (e.g. US 5.726,685). 
Furthermore, a so-called gesture input device is known from the publication US 5,617,312 In 
which the gestures of a person are recorded by a camera and are subsequently converted into 
corresponding control signals for contTOlling a pointer symbol or selection infonnation displayed 
on the display unit However, since said input devices are based on relatively complex pattern 
recognition, they are usually prone to errors and require large amounts of computing power. 
[0010] Known from the publication EP-A-773 434 is an input device for computer systems 
having a recording unit for serially recording Image Infonnation; an image evaluatiort unit for 
evaluating the recorded image infonnation and determining control information; a data 
processing unit for processing the conto-ol infonnation in selection information; and a display unit 
for displaying the selection infonnation. The Image evaluation unit detennines the control 
information from a relative displacement of ttie serially recorded image Infonnation. 

[0011] A device and a method for controlling a display Is also known from publication 
JP 10 240 436 A. in which a picture or menu displayed on the display is displaced vertically or 
horizontally respectively depending on a relative movement or displacement of a CCD camera. 
SUMMARY OF THE INVENTION 

[00121 In contrast, one possible object of the invention is to create an input device and a 
pertaining method for computer systems which Is space-saving, cost-effective and reliable. 
[001 3J Particularly on mobile videophones, the use of an image"evaluation unit in particular, 
which uses an image compression device for generating motion vectors, with the control 
infomnation being derived fi-om a main motion vector, enables the recording unit already present 
as well as the associated image compression method to be used simultaneously as an input 
device. 

[0014] Respective parallel displacement components of the Image information in the x and y 
directions are preferably used to generate the x and y components of the control infemiation, 
which enables conbiol of a pointer symbol or selection information to be realized on a two- 



PA6E 14/22' RCVD AT imOS 2:56:49 PM [Eastern Daylight rone] * SVR:USPT0-EF1(RF-6I26' DNIS:27383(I0 ' CSID:^' DURATION (min-ss):04-50 



Oct-04-05 02:43piD From-STAAS & HALSEY + T-523 P. 015/022 F-654 

3 

Docket No. 1454.1125 
Inventors: Gerhard BOCK et al. 

dimensional display. The control infonnation may however also have a z component which is 
produced fniHTi a concentric displacement of the image information in the x and y directions, 
which enables control of a pointer symbol or selection information to be realized on a three- 
dimensional or quasi three-dimensional display. In addition, this enables an intuitive 
enlargement or reduction of the display scale. To produce such a concentric displacement of 
Image information for three-dimensional input, the recording unit can preferably have a zoom 
function unit 

[001 5J A mobile multimedia communication tenninal is preferably used as the computer system, 
with the recording unit in a mobile videophone being used both for recording a call partner and 
as an input device. Analogously, a display unit is used both for displaying a call partner and for 
displaying an Internet page, organi2er page, etc. An extremely cost-effective and space-saving 
realization of the input device is thus achieved. The recording unit comprises in particular a 
CCD or CMOS sensor camera, which reduces the space requirement stf II further. 

10016] The recording unit preferably records image Information for input contre)l in the display 
direction of the display unit, but it can also record image information in the opposite direction. 
This is particulariy advantageous when the input device is provided In a mobile videophone, 
because when the videophone is placed on a suitable surface, by moving the videophone to 
and fro, an associated pointer symbol can be con-espondlngly displaced on the display unit in 
the same way as with a mouse input device. 

[0017] The recording unit preferably has a macro camera system for recording image 
information frewn a very close distance, which enables, for example, an input to be made by 
placing a finger directly on the reconjing unit If the recording unit additiwially has a pressure 
sensor for confimiing displayed selection fnfonmation. then the input device can be operated in 
the same way as a touch-screen input device. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[00181 These and other objects and advantages of the present invention will become more 
apparent and more readily appreciated from the following description of the prefen^d 
embodiments, taken in conjunction with the accompanying drawings of which: 

Fig. 1 shows a schematic diagram of the input device according to one aspect of the 
invention; 
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Rg. 2 shows a schematic diagram of serially reoorded Image Information to illustrate 
three-dimensional control; 

Figs. 3a and 3b show schematic diagrams of a mobile communication terminal having 
an input device according to a first exemplary embodiment; 

Fig. 4 shows a schematic diagram-of a mobile communication terminal having an Input 
device according to a second exemplary embodiment; and 

Figs, 5a to 5c show schematic diagrams of parts of a mobile communication terminal 
having an input device according to a thind exemplary embodiment 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

[001 9] Reference will now be made In detail to the preferred embodiments of the present 
invention, examples of which are illustrated in the accompanying dravwngs, wherein like 
reference numerals refer to like elements throughout 

[0020] Rg. 1 shows a schematic diagram of an input device. In Fig. 1. reference numeraM 
denotes a recording unit for serially recording image infomiation, wherein first Image information 
A essentially comprises a landscape with a house, two trees and an industrial building. After 
swiveling the reconding unit 1 1n the negative y direction, second Image information A' is 
recorded, which essentially comprises the house, the tree and the Industrial building of the first 
image Infomiation A plus an additional fence. Owing to the swiveling of the recording unit 1 , the 
additional tree present in the first image infomiation A Is not present in the second image 
information A*. 

[0021] An image evaluation unit 3 (AW) senses to evaluate the serially recorded first and 
second image infonnation A and A", with control information x. y and z being detemiined. 
Evaluation of the image evaluation unit 3 is essentially based here on determining a relative 
displacement between the serially recorded image infonnation. ie. between the first image 
Infomiation A and the second image information A". To be more precise, a main displacement 
vector HV which con^ponds to a displacement of the main image components within image 
information A and A' is detemnined. 

[0022] Given the swiveling of the recording unit 1 in the y direction illustrated In Fig. 1, when 
the Image Information A and A' is evaluated in the evaluation unit 3, the main displacement 
vector HV is accordingly detemiined, which essentially represents a displacement in the y 
direction. The control information In the y direction thus obtained is subsequently processed in 
a data processing unit 4 (DV) in such a way that a display unit 2 connected to the data 



PAGE 16122 ' RCVDAT mm 2:56:49 PM [Eastern Daylight Fime] ' SVR:USPTO-EF](RF-6/26' DNIS:27383(I(I ' CSID:^' DURATION (niin-ss):0(-50 



qct-04-05 02:44pin Frora-STAAS i HALSEY + T-523 P. 017/022 F-654 

5 

Docket No. 1454.1125 
Inventors: Gerhard BOCK et aL 

processing unit 4 causes a con^spondlng displacement of selection information P displayed on 
the display unit 2 to P". In Fig. 1 , the selection information displayed on the display unit 2 Is a 
pointer symbol P, which is located in an upper position for a first camera position for recording 
the first image information A. and moves to a lower position (P) when the recording unrt 1 1s 
swiveled to record the second image information A*. 

[0023] In accortlance with Fig. 1, therefore, a relative displacement of serially recorded Image 
Information can be produced and evaluated by swiveling a recording unit 1. which information 
produces a displacement of the displayed pointer symbols P to P' on the display unit 2. 

[0024] The relative displacement is determined here on the basis of the principal structures of 
image infomnation A and A'. To be more precise, the relative displacement is not determined on 
the basis of micro displacements, but rather on the basts of macro displacements in the Image 
Infomnation A and A'. Macn^ displacements are understood to be, for example, displacements 
of the general background of the image Infomiation A and A\ ymte micro displacements are 
displacements of limited image areas as a result of quick and instantaneous movements. 
Accordingly, a car moving through the landscape illustrated in Fig. 1 would not cause a macro 
displacement, but only a micnD displacement of Image information which is ignored by the input 
device. 

[0025] Only macro displacements of the recorded image infomriatfon are taken into account for 
the Inputting of control information or the control of selection Information P and P' displayed on 
the display unit 2, witfi a threshold decision element (not shown) setting an absolute magnitude 
for the macHD displacement. Accordingly, a macro displacement is taken into account by the 
evaluation unit 3 if, for example, at least go% of the image Information recorded by the 
recording unit 1 exhibits the same or a similar displacement. A parallel displacement in 
essentially the same x and y direction Is interpreted here the same displacement, while a 
displacement in the z direction is interpreted as a similar displacement 

[0026] Fig. 2 shows a schematic diagram for illustrating a relative displacement of image 
infonr^tion in a z direction. According to Fig. 2. the image information recorded by the 
reconjlng unit 1 comprises a house which is Initially at a distance (Image infbnnation A) and the 
same house which, as a result of coming closer in the z direction, is displayed as enlarged by 
the image infomiation A'. 

[0027] Accordingly, given a displacement in the 2 direction, the displacement vectors are not 
ordered parallel to one another in tiie x or y direction, but exhibit a concentric displacement 
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direction. Such concentrically oriented displacement vectors can therefore be evaluated by the 
evaluation unit 3 as control information in the z direction, thus enabling even three-dimensional 
control of the selection Information displayed on the display unit 2, i.e- pointer symbols P and P'. 
The display unit 2 may also be a three-dimensional display unit here (e.g. holographic display 
unit). 

[0028] As an ahemative to moving the recording unit 1 in the z direction, i.e, moving closer to 
the object to be recorded, the recording unit 1 can also have an (electronic or mechanical) zoom 
function unit (not shown) with which it is likewise also possible to perform a concentric 
displacement of the image infomiation. The recording unit 1 may also have a rotating or 
swiveling unit (not shown) with which the macro displacements in the x and y directions can be 
realized. 

[0029] Figs. 3a and 3b show schematic diagrams of a mobile multimedia communication 
tenmina! having the input device according to a first exemplary embodiment of the invention. In 
Fig. 3a. MM denotes a mobile multimedia communication temiinal, as may be realized for 
example in the form of a mobile videophone with Intemet access. In addition to the usual 
function units, the mobile multimedia communication terminal MM essentially has a recording 
unit 1 , a display unit 2 and an acfivation button T for activating/deactivating the input device. In 
the case of an Intemet access, the enlarged Intemet page IS (web page) Is displayed on the 
display unit 2 for example, with a pointer symbol P being displayed as selechon information at 
the position I. To move the pointer symbol P from position I to position II, according to Fig, 3b a 
user can, for example, move the mobile multimedia communication tenninal MM downward in 
front of his face, with the result that, instead of image infomfiation A at position I, the image 
information A' at position II is now recorded by the recording unrt 1 . The resulting macro 
displacements are detected by the evaluation unit 3, as desoTbed above, and are processed by 
the data processing unit 4 in such a way tiiat the display unit 2 now displays the pointer P at 
position II of the displayed intemet page IS. A displacement of the pointer symbol P showing 
the selection infonnation to P' is thus achieved in this way by displacing the recording unit 1 
relative to the recorded background. 

[0030] As was already described above, it is also possible to perform a swiveling instead of 
the vertical displacement of the mobile multimedia communication tenminal from a position I to a 
position II, which enable the pointer symbol P to be displaced in the same manner to position II 
for the pointer symbol P'. 
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[0031] Mobile multimedia communication temrimals, as lllustrarted by way of example in Rgs. 
3a and 3b, are especially advantageous if they are equipped as mobile videophones. Mobile 
videophones already have a large display unit 2 as well as a recording unit 1 for recording 
image infomnation. Moreover, sudi mobile videophones already have an (mage compression 
device as image evaluation unit 3 for compressing the recorded image infomnation. The 
algorithms used for image compression detect a movement in the image infbmriation A and A\ 
with the movement of the total image information being calculated in such a way to produce a 
total or main motion vector which is equivalent to the movement or displacement of the mobile 
multimedia communication terminal MM or the associated Image infomiation A and A' 
respectively. This total or main motion vector, which can be readily calculated during image 
compression, can now be processed as control infonnation with corresponding x» y and z 
components, and displace the selection inforrration, i.e. the pointer symbols P or P\ 
accordingly. Since the compression algorithms used are essentially already present in a mobile 
videophone, no additional costs are incun^ed. To switch over or activate/deactJvate the input 
device, the mobile videophone or mobile multimedia communication terminal MM requires only 
an activation button T, 

[0032] Fig. 4 shows a schematic diagram of a mobile multimedia communication tenninal 
having an input device according to a second exemplary embodiment In Fig. 4 the same 
reference symbols denote the same or con^esponding components as in Figs. 3a and 3b, so a 
detailed description is dispensed with below, 

[0033] On a mobile multimedia communication tenninal MM having an Input device according to 
the second exemplary embodiment, the display unit 2 does not display the complete Internet 
page !S. but only a selected section F. This way of displaying of information, for example finom 
file Internet or from another application, has the advantage=that the information on the displayed 
page IS is more readily legible for the user. 

10034] According to the second exemplary embodiment, the input device now has intuitive 
control of the information page IS to be displayed. To be mora precise, a user need only 
displace the mobile multimedia communication terminal with the recording unit 1 in the x and y 
directions in order to reach the respective sections of the information page IS, so that the 
information page IS can be read intuitively (i.e. by displacing a viewing window) without using 
any further keys or input media. 
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[0035] In addition, a displacenrient in the z direction, I.e. a fonvard/backward movement of the 
mobile multimedia communication terminal MM, can enlarge or reduce the size of the selected 
section F, which enables intuitive use comparable to the use of a magnifying glass. 

[0036] According to Figs. 3 and 4, the recording unit 1 is located on the front of the mobile 
multimedia communication temninal MM. It can however also be located on the rear of the 
terminal, or a second recording unit may be an^nged on the rear of the mobile multimedia 
communication terminal MM. Given suitable realis^ation of the optics, i.e. the ability to focus the 
recording unit 1 even at a very short distance, the mobile multimedia communication temiinal 
MM can also be placed rear down on a suitably structured surface and the whole unit can be 
displaced in the same way as a mouse. The recording unit located on the rear detects here, as 
described above, the relative movement or relative displacement of the recorded Image 
information (structured surface) and determines from it the corresponding control information for 
controlling the selection information, i.e. a pointer symbol P or a display section F of the 
information page IS. This results In the input device being particularly easy to operate, since it 
essentially corresponds to a mouse input device. In this case the mobile multimedia 
communication tenninal preferably has the selection buttons (not shown) known from mouse 
input devices. 

[0037] Figs. 5a to 5c show enlarged partial views of a mobile multimedia communication 
terminal MM having an input device according to a third exemplary embodiment. In Figs. 5a to 
5c. the same reference symbols denote the same components as in Figs. 1 to 4, so a det^Ied 
description is dispensed with below. 

[0038] According to Figs. 5a to 5c, the input device according to the third exemplary 
embodiment can be used in the same way as a so-called "touch-pad" input device. In Fig. 5a, 
reference numeral 1 again denotes a recc^ding unit 1, but 'One which is a spedal macro camera 
system. To be more precise, the recording unit 1 according to Figs. 5a to 5c is al>le to record 
very fine structures in the immediate vicinity of the camera, by which, for example, the finger 
lines of a finger placed directly on the recording unit are recorded in sharp focus. 

[0039] According to Figs. 5a to 5c, the display unit 2 shows a telephone directory list with a 
plurality of nan^ and conresponding phone numbers. The telephone directory list displayed on 
the display unit 2 thus con^spwds to an infomnatton page IS, with the selection Information 
being represented neither by a pointer symbol P for Indicating a selection position, nor by a 
display area F, but by a predetermined selection field B in the form of a, for example, colored 
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bar To control the input, or to select the name Meier with phone number 345678, the user then 
places his finger on the recording unit 1 , with the input device activated by the activation button 
T (not shown). According to Fig. 5b, the finger lines are now recorded as image information A, 
and the user moves his finger downward to displace the bar B. Fig, 5c shows here a partial 
view of the mobile multimedia communicatbn terminal MM, with the finger shown at a lower 
position. The recording unit 1 accordingly detected a displacement of the recorded image 
information A A* on the basis of the finger lines and evaluated this as a vertical displacement 
downward- The evaluation unit 3 accordingly outputs corresponding control information to the 
data processing unit 4, which results in the bar B being displaced downward on the display unit 
2 and now highlighting the desired name Meier with phone number 345678, 

[0040] To select the desired name, according to the third exemplary embodiment ttie input 
device can also have a pressure sensor for confinning the displayed selection infbnnation, i.e, 
the name highlighted by the bar B, which dispenses with the need for ftjrther fixed keys or so- 
called menu-driven "soft keys". 

[0041] Given suitable dimensioning of the recording unit 1 and the evaluation unit 3, in this way 
it is also possible to make an input in a way known from '"touch pads" and/or *^ouch screens" for 
example. The Integration of a pressure sensor In the recording unit 1 further reduces the space 
required for the input device and enhances ease of operation. In comparison with known "touch 
pads" in particular, with the input device according to the third exemplary embodiment. It is not 
the main focus of a surface area which is determined, but rather the exact position or movement 
of the finger Is tracked on the basis of the lines present on it. The precision of this input devfce 
is therefore much greater than that of known input devices. With the input device according to 
the third exemplaiy embodiment, therefore, essentially a relative macro displacement of the 
recorded image information is also used to control selection information. 

[0042] A CCD or CMOS sensor camera is preferably used as the recording unit 1 . The 
recording unit 1 is located here either on the front or on the rear of an input or computer system 
respectively, but it is not limited thereto and may also be a^anged at a different position and be 
moved around, for example by min-ors, in the same or opposite direction to the display direction 
of the display unit 2. 

[0043] The invention was described above with reference to a mobile multimedia 
communication terminal. It is however not limited thereto, and like>Arise encompasses ultra* 
small computer systems without a communication link, such as palmtop computers for example. 
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Likewise, the present invention may also be used on corded telecommunfcation tentiinals and 
other computer systems. 

[0044] The Invention has been described in detail with particular reference to preferred 
embodiments thereof and examples, but it will be understood that variations and modifications 
can be effected wifliin the spirit and scope of the invention. 
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